Exemplary embodiments relate generally to training of manual tasks, and more particularly, to real-time in-situ feedback of task performance relating to physical examinations.
Across fields such as mechanical, manufacturing and industrial engineering, and medicine, training of manual tasks which require learning both psychomotor (i.e., coordinating muscle movements) and cognitive (e.g., spatial layout, search) components is often accomplished by pairing the learner with an expert (“learner-expert” model). The expert observes the learner performing the task and provides the learner with feedback. Though often the standard, drawbacks of the “learner-expert” model include restricting a student's practice to times the student can be observed (conversely, placing demands on an expert's time) and the ability of experts to provide only inherently subjective feedback of varying granularity (e.g., a clinician can provide a medical student with more detailed feedback in a laparoscopic procedure than in a prostate exam, because the laparoscope allows the expert to view the student's actions within the patient's body; this is not the case for a prostate exam). To provide additional learning opportunities, mixed environments (MEs) for learning have been proposed, in which the role of the expert is assumed by visual information overlaid on the physical objects being manipulated.
MEs have the potential to benefit learning joint psychomotor-cognitive tasks. For example, MEs provide physical manipulation simultaneous with guidance for learning psychomotor and cognitive tasks. MEs may incorporate sensors (e.g., tracking) to provide automated, objective, quantitative evaluations of learner performance. Additionally, this sensor data can be processed to provide constructive, real-time feedback of performance (particularly feedback which is unavailable in a purely physical environment). Constructive feedback of learner performance is known to improve learning of cognitive, psychomotor, and joint psychomotor-cognitive tasks in physical environments. Also, MEs can integrate this automated learner evaluation and feedback in an on-demand learning environment.
Although MEs have many potential benefits for learning joint psychomotor-cognitive tasks, there are substantial challenges to realizing these benefits. The challenges to ME developers include: (1) Overcome accuracy and latency limitations of tracking in order to accurately sense the learner's psychomotor actions in real-time. Real-time sensing of learner actions is required to evaluate psychomotor task performance, and as more fine-motor skills are involved, the sensing approach required becomes more complex. In these cases, using only fiducial-based or glove-based tracking becomes less feasible due to the need to track complex hand and finger movement without encumbering this movement. (2) In real-time, extract quantitative measures of learner task performance from this noisy and possibly high-dimensional sensor data. (3) Provide real-time feedback of this performance data in a form meaningful to the learner.
Common approaches may avoid some of the pitfalls represented by these challenges, but do not fully realize all potential benefits, by: (1) Focusing on only the cognitive components of the task. These MEs provide feedback to guide cognitive tasks by presenting a priori acquired information, e.g., spatial relationships among task elements. Without providing feedback of psychomotor components, real-time sensing of learner actions is not required. However, intuitively, this reduces the ME's effectiveness in training tasks requiring psychomotor skills not already well developed in the learner. (2) Having human experts process captured performance data into post-experiential feedback. In this approach, real-time sensing of learner actions and evaluation of learner psychomotor performance is not required. However, reliance on offline expert analysis of learner performance lengthens the learning process and restricts the ability of the ME to provide on-demand learning opportunities.
MEs have successfully been applied to train joint psychomotor-cognitive tasks in which the psychomotor components are previously well developed in the learner, such as assembly tasks. MEs facilitate learning of assembly tasks by providing guidance through, e.g., a static reference diagram of the assembly or step-by-step instructions (i.e., arrows indicating where to place the next component). Although these cognitive aids reduce learners' errors and cognitive load, it has not been demonstrated that the psychomotor skills required for assembly tasks, e.g., reaching, grasping, aiming, and inserting are learned within the ME or are well developed in learners prior to training with the ME. Similar approaches to presenting cognitive aids for spatial tasks have been applied in MEs for learning printer and aircraft maintenance. MEs also aided developing cognitive models of gas flow within anesthesia machines. Similarly to the MEs for learning assembly, the psychomotor skills used were simple, e.g., pressing buttons to alter gas flow, and were not a focus of the ME.
Learning of psychomotor skills in virtual environments (VEs) and MEs has been demonstrated for two classes of tasks: tasks in which the learner uses only his or her body to interact with a virtual environment (e.g., tai chi, tennis), and tasks which focus on the interaction between a few physical objects (manipulated by the user) and a virtual environment (e.g., rehabilitation, use of forceps in birthing, laparoscopy). These systems train psychomotor tasks using the paradigm of mimicking a virtual expert (presented as pre-recorded video or a 3D reconstruction), occasionally augmented with visualization of limb trajectories. Learner performance data is captured by tracking a small number of passive props, instrumented interface devices, or is generated offline by expert review. Incorporation of real-time feedback in these systems is rare and typically replicates feedback available in the physical learning environments for these tasks, e.g., patient bleeding in laparoscopic surgery.
In order to provide real-time feedback of performance, the feedback must be a function of the learner's performance measured in real-time. This contrasts with the approach of augmenting the environment with visual elements computed from a priori measured information, e.g., as used in labeling and “x-ray” vision for navigation in outdoor environments, and assembly and maintenance learning environments. Vision-based techniques are used to augment printed images with information a priori embedded in the printed image.
What is needed is a way to provide real-time in-situ feedback of learner performance that resolves the above-described challenges while realizing the benefits of MEs for learning joint psychomotor-cognitive tasks.